Robotic surgical safety via video processing

ABSTRACT

One example method for improving robotic surgical safety via video processing includes identifying, during a robotic surgical procedure, one or more surgical tools using one or more images of the surgical procedure captured by a camera, the robotic surgical procedure employing a robotic surgical device controlling the one or more identified surgical tools; predicting, for at least one of the one or more images, one or more loaded surgical tools that are controlled by the robotic surgical device and should be in a field of view of the camera; comparing the one or more identified surgical tools with the one or more loaded surgical tools; determining that at least one of the one or more loaded surgical tools does not match any of the one or more identified surgical tools; and causing the at least one loaded tool of the robotic surgical device to be disabled.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.16/694,223, filed Nov. 25, 2019, which claims priority to U.S. PatentApplication No. 62/775,501, filed Dec. 5, 2018, titled “Robotic SurgicalSafety via Video Processing,” the entirety of which is herebyincorporated by reference.

FIELD

The present application generally relates to robotic surgery and videoprocessing, and more particularly relates to improving the safety ofrobotic surgery using video processing.

BACKGROUND

Robotic surgeries are becoming increasingly popular nowadays because oftheir advantages over the traditional human-operated open surgeries.Surgical tools utilized in robotic surgeries have improved levels ofdexterity over a human surgeon. These tools can provide the surgeonmaximum range of motion and precision. In addition, high-definitioncameras associated with the surgical tools can provide a better view ofthe operating site to the surgeon than human eyes can provide. Further,the small size of the robotic surgical tools allows the surgeries to bedone in a minimally invasive manner thereby causing less trauma on thepatient's body.

To ensure safety during a surgery, it is important for a surgeon to beable to visually monitor the surgical tools he is using. However, it isa non-trivial task to calculate the positions of the surgical toolsrelative to the camera, considering that the surgical tools and thecamera are typically attached to the distal ends of long, semi-rigidrobot arms. Small deflections or errors in joint measurements can becomerelatively large errors in predicted tool positions. Additionally, evenif the tools are within the camera view and visible to the surgeon,because there is no haptic feedback from the surgical tools to thesurgeon, it is possible for the surgeon to inadvertently apply too muchforce to patient anatomy without realizing it, causing unintentionalinjury to the patient.

SUMMARY

Various examples are described for improving robotic surgical safety viavideo processing. One example method includes identifying, during arobotic surgical procedure, one or more surgical tools using one or moreimages of the surgical procedure captured by a camera, the roboticsurgical procedure employing a robotic surgical device controlling theone or more identified surgical tools; predicting, for at least one ofthe one or more images, one or more loaded surgical tools that arecontrolled by the robotic surgical device and should be in a field ofview of the camera; comparing the one or more identified surgical toolswith the one or more loaded surgical tools; determining that at leastone of the one or more loaded surgical tools does not match any of theone or more identified surgical tools; and causing the at least oneloaded tool of the robotic surgical device to be disabled.

Another method includes identifying a surgical tool used by a roboticsurgical device in a surgical procedure using one or more images of thesurgical procedure; estimating a pose of the surgical tool based, atleast in part, upon the one or more images; determining a differencebetween the estimated pose of the surgical tool and a predicted pose ofthe surgical tool predicted through a kinematic chain model of therobotic surgical device; calculating a force exerted by the surgicaltool based, at least in part, upon the difference of the estimated poseand predicted pose of the surgical tool; determining that the force ishigher than a threshold; and causing the force of the surgical tool tobe reduced.

One example robotic surgical device includes a processor; a camera; anda non-transitory computer-readable medium having processor-executableinstructions stored thereupon, which, when executed by the processor,cause the processor to: identify, using one or more images of a surgicalprocedure captured by the camera during a robotic surgical procedure,one or more surgical tools used in the surgical procedure; predict, forat least one of the one or more images, one or more loaded surgicaltools that are loaded during the surgical procedure and should be in afield of view of the camera; compare the one or more identified surgicaltools with the one or more loaded surgical tools; determine that atleast one of the one or more loaded surgical tools does not match any ofthe one or more identified surgical tools; and cause the at least oneloaded surgical tool of the robotic surgical device to be disabled.

An example non-transitory computer-readable medium comprisingprocessor-executable instructions to cause a processor to identify,during a robotic surgical procedure, one or more surgical tools usingone or more images of the surgical procedure captured by a camera, therobotic surgical procedure employing a robotic surgical devicecontrolling the one or more identified surgical tools; predict, for atleast one of the one or more images, one or more loaded surgical toolsthat are controlled by the robotic surgical device and should be in afield of view of the camera; compare the one or more identified surgicaltools with the one or more loaded surgical tools; determine that atleast one of the one or more loaded surgical tools does not match any ofthe one or more identified surgical tools; and cause the at least oneloaded surgical tool of the robotic surgical device to be disabled.

Another example non-transitory computer-readable medium comprisingprocessor-executable instructions to cause a processor to identify asurgical tool used by a robotic surgical device in a surgical procedureusing one or more images of the surgical procedure; estimate a pose ofthe surgical tool based, at least in part, upon the one or more images;determine a difference between the estimated pose of the surgical tooland a predicted pose of the surgical tool predicted through a kinematicchain model of the robotic surgical device; calculate a force exerted bythe surgical tool based, at least in part, upon the difference of theestimated pose and predicted pose of the surgical tool; determine thatthe force is higher than a threshold; and cause the force of thesurgical tool to be reduced.

These illustrative examples are mentioned not to limit or define thescope of this disclosure, but rather to provide examples to aidunderstanding thereof. Illustrative examples are discussed in theDetailed Description, which provides further description. Advantagesoffered by various examples may be further understood by examining thisspecification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate one or more certain examples and,together with the description of the example, serve to explain theprinciples and implementations of the certain examples.

FIG. 1 shows an example of a robotic surgery system with improved safetyvia video processing;

FIG. 2 shows an example kinematic chain model of a robotic arm holding asurgical tool that can be utilized to predict the position of thesurgical tool;

FIGS. 3A and 3B show example safety issues that can be addressed by therobotic surgery system with improved safety via video processing;

FIG. 4 shows a block diagram illustrating aspects of the safety moduleof the robotic surgery system with improved safety via video processing;

FIGS. 5A-D show examples of simulated images for matching identifiedsurgical tools with loaded surgical tools to identify safety issuesassociated with the robotic surgical device;

FIG. 6 shows an example method for improving robotic surgical safety viavideo processing;

FIG. 7 shows an example method for estimating tool force in a roboticsurgery system with improved safety via video processing; and

FIG. 8 shows an example computing device suitable for implementingaspects of the techniques and technologies presented herein.

DETAILED DESCRIPTION

Examples are described herein in the context of improving roboticsurgical safety via video processing. Those of ordinary skill in the artwill realize that the following description is illustrative only and isnot intended to be in any way limiting. Reference will now be made indetail to implementations of examples as illustrated in the accompanyingdrawings. The same reference indicators will be used throughout thedrawings and the following description to refer to the same or likeitems.

In the interest of clarity, not all of the routine features of theexamples described herein are shown and described. It will, of course,be appreciated that in the development of any such actualimplementation, numerous implementation-specific decisions must be madein order to achieve the developer's specific goals, such as compliancewith application- and business-related constraints, and that thesespecific goals will vary from one implementation to another and from onedeveloper to another.

In an illustrative example of a robotic surgery system with improvedsafety via video processing, one or more surgical tools are loaded by arobotic surgical device using respective robotic arms. A surgeon canoperate these surgical tools through controls at a surgeon console. Acamera is also loaded by the robotic surgical device to capture imagesor videos of a surgical procedure performed using the loaded surgicaltools.

The position of a loaded surgical tool relative to the camera can bepredicted using a kinematic model of the robotic arm holding thesurgical tool and a kinematic model of the robotic arm holding thecamera. This relative position can be utilized to determine whether acertain loaded tool is in the field of view of the camera. If so, itmeans that the surgeon can see the loaded tool in camera and thus itshould be enabled to allow the surgeon to use it. If the loaded surgicaltool is predicted to be outside the camera view, it would be unsafe toallow the surgeon to use it and thus it should be disabled.

The technology presented herein improves the safety of the roboticsurgery system by adding an additional mechanism for identifying theposition of the surgical tool and by estimating the force exerted by thesurgical tool based on the estimated tool position. Predicting thelocation of a surgical tool solely based on a kinematic analysis may beinsufficient or inaccurate in some cases because various factors cancause a calculated surgical tool position to be incorrect, such ascompliance within one or more joints, sensor tolerances within thekinematic chain, external forces applied at one or more locations alongthe kinematic chain that deflect a member of the chain, etc. Thus,examples according to this disclosure employ image analysis in additionto modelling a kinematic chain to determine whether a loaded surgicaltool is within a camera's field of view. In particular, apart frompredicting the positions of loaded tools using the kinematic chain modelof the robotic surgical device, the positions of the loaded surgicaltools can also be estimated by using videos of the surgical procedurecaptured by the camera. Images captured during a surgical procedure canbe analyzed to identify surgical tools that are in the field of view ofthe camera. The identification can include detecting the positions,poses, and types of the surgical tools captured in the surgical images.The information about the identified surgical tools can then be comparedwith the information of the loaded surgical tool(s) predicted to be inthe camera's field of view through the kinematic models at the time thesurgical images are captured to identify any discrepancies, anyexcessive forces applied by the surgical tool(s), or other safetyissues.

As used herein, surgical tools that are identified from the images arereferred to as “identified surgical tools” or “identified tools” and thepositions of the identified surgical tools estimated based on the imagesare referred to as “identified positions.” Surgical tools that aredetermined to be in the camera's field of view based on the kinematicchain models are referred to herein as “loaded tools” and the positionsof the loaded tools predicted through the kinematic chain models arereferred to as “predicted positions.”

In one implementation, the positions of the identified surgical toolsare compared with the predicted positions of the loaded surgical toolsto determine a one-to-one correspondence between the identified surgicaltools and the loaded surgical tools. It can then be determined whetherthe number of identified surgical tools in the surgical images is thesame as the number of the loaded surgical tools. If they are different,then the loaded surgical tools that are predicted to be in the cameraview but are not detected in the surgical images should be disabled. Onthe other hand, surgical tools that are predicted to be outside thecamera view, but are detected in the surgical images should be enabled.

For the surgical tools that have a one-to-one correspondence between theloaded surgical tools and the identified surgical tools, the type of aloaded surgical tool can be compared with the type of its correspondingidentified surgical tool. If the types of the two tools are different,then the surgical tool might be loaded erroneously and should bedisabled. If the types of the two tools are the same, a differencebetween the positions and poses of the two tools in a three-dimensionalspace can be calculated to estimate the force exerted by the surgicaltool. The estimated tool force can be compared with a force threshold todetermine if excessive force has been applied by the surgical tool. Ifso, a warning message can be generated to notify the surgeon to reducethe tool force.

This illustrative example is given to introduce the reader to thegeneral subject matter discussed herein and the disclosure is notlimited to this example. The following sections describe variousadditional non-limiting and non-exhaustive examples and examples ofimproving robotic surgical safety via video processing.

Referring now to FIG. 1 , FIG. 1 shows an example robotic surgicalsystem 100 with improved safety via video processing. The roboticsurgical system 100 can include a robotic surgical device 114 configuredto operate on a patient 130. The robotic surgical system 100 can alsoinclude a surgeon console 104 connected to the robotic surgical device114 and configured to be operated by a surgeon 102 in order to controland monitor the surgeries performed by the robotic surgical device 114.The robotic surgical system 100 might include additional stations (notshown in FIG. 1 ) that can be used by other personnel in the operatingroom, for example, to view surgery information, video, etc., sent fromthe robotic surgical device 114. The robotic surgical device 114, thesurgeon console 104 and other stations can be connected directly orthrough a network, such as a local-area network (“LAN”), a wide-areanetwork (“WAN”), the Internet, or any other networking topology known inthe art that connects the robotic surgical device 114, the surgeonconsole 104 and other stations.

The robotic surgical device 114 can be any suitable robotic system thatcan be used to perform surgical procedures on a patient. A roboticsurgical device 114 may have one or more robotic arms connected to abase. The robotic arms may be manipulated by a tool controller 116,which may include one or more user interface devices, such as joysticks,knobs, handles, or other rotatable or translatable devices to effectmovement of one or more of the robotic arms. The robotic arms may beequipped with one or more surgical tools to perform aspects of asurgical procedure. For example, the robotic arms may be equipped withsurgical tools 126A-126C, (which may be referred to herein individuallyas a surgical tool 126 or collectively as the surgical tools 126). Thesurgical tools 126 can include, but are not limited to, tools forgrasping for holding or retracting objects, such as forceps, graspersand retractors, tools for suturing and cutting, such as needle drivers,scalpels and scissors, and other tools that can be used during asurgery. Each of the surgical tools 126 can be controlled by the surgeon102 through the surgeon console 104 and the tool controller 116.

In addition, the robotic surgical device 114 may be equipped with one ormore cameras 128, such as an endoscope camera, configured to provide aview of the operating site to guide the surgeon 102 during the surgery.In some examples, the camera 128 can be attached to one of the roboticarms of the robotic surgical device 114 controlled by the toolcontroller 116 as shown in FIG. 1 . In other examples, the camera 128can be attached to a mechanical structure of the robotic surgical device114 that is separately from the robotic arms.

Different robotic surgical devices 114 may be configured for particulartypes of surgeries, such as cardiovascular surgeries, gastrointestinalsurgeries, gynecological surgeries, transplant surgeries,neurosurgeries, musculoskeletal surgeries, etc., while some may havemultiple different uses. As a result, different types of surgicalrobots, including those without robotic arms, such as for endoscopyprocedures, may be employed according to different examples. It shouldbe understood that while only one robotic surgical device 114 isdepicted, any suitable number of robotic surgical devices may beemployed within a robotic surgical system 100.

In some examples, robotic surgical devices (or a respective controller)may be configured to record data during a surgical procedure. Forexample, images and videos of the surgical procedures performed by therobotic surgical device 114 can also be recorded and stored for furtheruse. For instance, a datastore 124 can be employed by the roboticsurgical device 114 to store surgical images 132 of surgical procedurescaptured by the camera 128. The surgical images 132 can be stored asseparate images or as one or more video files.

In the example shown in FIG. 1 , surgical images 132 of a roboticsurgical procedure captured by the camera 128 can also be transmitted tothe surgeon console 104 and be displayed on a video monitor 108 in realtime so that the surgeon 102 can view the procedure while the surgicaltools 126 are being operated on the patient 130. In this example, thesurgeon 102 can use the surgeon console 104 to control the surgicaltools 126 and the camera 128. The surgeon 102 can use controls 106 onthe surgeon console 104 to maneuver the surgical tools 126 and camera128 by sending control signals 110 to the corresponding surgical tools126 or camera 128.

For example, the surgeon 102 can move the camera 128 to variouspositions of patient body to find the proper operating site. The surgeon102 can then move the camera 128 to a position where a surgical tool 126to be used is located to bring the surgical tool 126 inside the field ofview of the camera 128. The surgeon 102 may then move the surgical tool126 along with the camera 128 to the operating site. For safety reasons,the robotic surgical system 100 will prevent the surgeon 102 from movinga surgical tool 126 if the surgical tool 126 is not in field of view ofthe camera 128. As a result, if a surgical tool 126 lies outside thefield of view of the camera 128, the robotic surgical system 100 candisable the out-of-view surgical tool so that it cannot be moved oroperated. Such a tool may be re-enabled when it is back in the cameraview. Here, a tool is considered to be within the field of view of acamera if a substantial amount of the tool is within the field of viewof the camera. In other words, if only a small portion, such as the tipof the tool, is within the camera view, it is still unsafe to allow thesurgeon to operate the tool and thus the tool is not considered to bewithin the camera view. The amount of the tool that needs to be in thecamera view in order to allow safe operation might be different fordifferent types of tools.

Whether a surgical tool 126 is in the field of view of the camera 128can be determined by determining the position of the surgical tool 126relative to the camera 128. The robotic surgical device 114 can includea position predictor 118 to predict the position of the surgical tool126. The prediction can be performed through a kinematic chain model ofthe robotic arm of the robotic surgical device 114 holding the surgicaltool 126. The kinematic chain model can predict the position of thesurgical tool 126 based on the dimensions and connectivity of the linksand joints of the robotic arm. Information collected from varioussensors deployed in the robotic arm, such as the direction and degree ofa rotation occurred at a joint, can also be utilized to perform theprediction. In addition, computer aided design (CAD) models of thesurgical tool can also be utilized to predict the position of thesurgical tool.

FIG. 2 illustrates an example kinematic chain model for an robotic arm.In this example, the robotic arm includes three links, Link 1, Link 2and Link 3, having respective lengths L1, L2 and L3. The three links areconnected via two joints having respective angles θ₂ and θ₃. The base ofthe robotic arm and Link 1 form angle θ₁. If the base of the robotic armis used as the origin (0,0,0), the coordinates (X,Y,Z) of the positionof the tool tip in the three-dimensional space can be determined basedon the length of each of the three links, namely, L1, L2 and L3 and thevalues of angles θ₁, θ₂ and θ₃. Similarly, the position of the camera128 can be predicted by utilizing the kinematic chain model of therobotic arm holding the camera 128.

Once the positions and poses of the surgical tools 126 and the camera128 are determined, the relative position of each surgical tool 126 withregard to the camera 128 can also be determined. In addition, the fieldof view of the camera 128 can be determined based on the parameters andsettings of the camera 128, such as the lens and zooming information.Based on these types of information, it can be determined whether asurgical tool 126 is inside the field of view of the camera 128 and thusdetermining whether the surgical tools 126 should be disabled or not.The position predictor 118 can store the predicted position of thesurgical tools 126 as well as other information about the surgical toolsloaded by the tool controller 116, such as the type of each loaded tool,as the loaded tool information 122.

However, various errors, such as compliance errors and measurementerrors, can cause the prediction of the position or pose of the surgicaltools 126 and/or the camera 128 to be inaccurate. This problem isespecially prominent when the robotic surgical device 114 have longrobotic arms. In those cases, small deflections or errors in proximaljoint measurements can become relatively large errors in predictedposition or pose for the surgical tool 126 or the camera 128. As aresult, a surgical tool 126 that is predicted to be inside the field ofview of the camera 128 might be actually outside the camera view.Allowing the surgeon 102 to move or operate such a tool can causeunintentional injury to the patient 130. In addition, a surgical tool126 that is predicted to be outside the field of view of the camera 128might actually be visible through the camera 128. Disabling such anin-view surgical tool can cause frustration and confusion to the surgeon102 since he is able to see the tool but not control it.

FIG. 3A illustrates one example of the problem caused by inaccurateposition prediction of the surgical tool 126. In FIG. 3A, an out-of-viewsurgical tool is erroneously predicted to be inside the camera view. Thesurgeon 102 is able to operate the out-of-view tool, but unable to seeit in the surgical images 132 displayed on the video monitor 108. As aresult, the out-of-view surgical tool might damage the tissue or organthat is also outside the field of view of the camera 128 without thesurgeon 102 realizing it.

Furthermore, even if a tool is in the field of view of the camera 128,it is difficult for the surgeon 102 to know the amount of force appliedby the surgical tool 126 to the patient's anatomy. Excessive amount offorce applied on the patent 130 can cause deflection on the surgicaltool 126 thereby resulting in the actual position of the surgical tool126 to be different from the predicted position. FIG. 3B illustrates anexample of this problem. In FIG. 3B, an in-view surgical tool is blockedby an out-of-view anatomy. The force applied by the surgical tool on theout-of-view anatomy deflects the surgical tool causing its positiondeviating from its predicted position. If the force deflecting the toolis large, it can cause injury to the patient 130. For example, theapplied force may injure the out-of-view anatomy or it may causesignificant deflection in the tool leading to undesired movement of thetool within the patient 130.

To improve the safety of robotic surgeries and to address the aboveproblems, the robotic surgical device 114 can further include a safetymodule 120 according to the technologies disclosed herein. The safetymodule 120 can analyze the surgical images 132 captured during thesurgical procedure to identify the position and pose of the surgicaltools appeared in the surgical images 132. For example, stereo imagestaken by surgical endoscopes, which are typically stereo cameras, can beutilized to estimate depth information of the surgical tools moreaccurately thereby facilitating the detection of the position and poseof the surgical tool. Such information can be compared with the positioninformation provided by the position predictor 118 to identify safetyissues in the surgical procedure and provide safety instructions 136 tothe robotic surgical device 114 to address those safety issues.

As used herein, the surgical images 132 can include one or more videosof the surgical procedure captured by the camera 128 or any individualimages contained in such videos. The analysis can include identifyingsurgical tools 126 used in the surgical procedure from the surgicalimages 132, for example, through object recognition or other image/videoprocessing techniques. The types of the identified surgical tools 134can also be determined. The position and type information of theidentified surgical tools 134 can then be compared with the loaded toolinformation 122 obtained from the position predictor 118 thatcorresponds to the analyzed surgical images 132 to determine if theidentified surgical tools match the loaded surgical tools.

A match can be determined if the number and the types of the identifiedsurgical tools are the same as the number and types of the loadedsurgical tools as specified in the loaded tool information 122. If thenumber of the identified surgical tools is different from the number ofthe loaded surgical tools, the safety module 120 can further identifythe loaded surgical tools that are not in the identified surgical tools.These tools represent the surgical tools that are predicted to be in thefield of view of the camera, but missing from the surgical images 132.As discussed above, these out-of-view loaded surgical tools can causedamages to the patient and thus should be disabled. The safety module120 can instruct the tool controller 116 to disable the out-of-viewsurgical tools. Disabling a surgical tool 126 can include, but is notlimited to, preventing the input of the surgeon 102 from being sent tothe surgical tool 126; deactivating energy applied by the surgical tool126, e.g., for cauterizing tissue, locking the drive motors or joints toprevent movement of the surgical tool 126, and so on. Alternatively, oradditionally, the safety module 120 can slow down the out-of-viewsurgical tools to reduce the risk of the injury caused by these tools.

Additionally, the safety module 120 can further determine surgical toolsthat are in the identified surgical tools, but not in the loadedsurgical tools. These are the surgical tools that are predicted to beoutside the field of view of the camera, but are visible in the surgicalimages 132. The safety module 120 can allow these surgical tools to beenabled, for example, by instructing the tool controller 116 to removeany mechanisms implemented when disabling the surgical tools so thatthey can be used by the surgeon 102 when needed. For example, enabling asurgical tool can include allowing the input of the surgeon 102 be sentto the surgical tool 126, activating energy applied by the surgical toolif necessary, unlocking the drive motors or joints to allow movement ofthe surgical tool and so on.

If the number of the tools in the loaded surgical tools and theidentified surgical tools are the same, but the types of the surgicaltools are different, it can be determined that a tool with a wrong typemight have been loaded and used in the surgical procedure. The safetymodule 120 can similarly instruct the tool controller 116 to disable theunmatched surgical tools 126 to prevent injury or avoid further damage.

Even if the number and the types of the loaded surgical tools matchthose of the identified surgical tools, it is still possible that asurgical tool 126 is at a position different from its predictedposition. Such a positional difference can be caused by the forceexerted by the surgical tool 126 on patient issue deflecting thesurgical tool 126. In those scenarios, the force of a surgical tool 126can be estimated by the safety module 120 to determine whether the toolforce is excessive and unsafe to the patient 130.

The tool force can be estimated by evaluating the difference between thepositions and poses of the identified surgical tool and thecorresponding loaded surgical tool. The difference in tool pose alongwith information such as the shape and materials of the tool and therobotic arm can be utilized to estimate the tool force. The estimatedtool force can then be compared with a threshold value to determine ifthe force of the tool is excessive. If so, the safety module 120 cancause a warning message to be generated and transmitted to the surgeon102 so that the surgeon 102 can take proper actions, such as moving thesurgical tool to a different location, changing the pose of the surgicaltool, or even retracting the surgical tool. Alternatively, oradditionally, the safety module 120 can send a safety instruction to thetool controller 116 to move the surgical tool in a non-intrudingposition before disabling it. Additional details concerning theoperations of the safety module 120 are provided below with regard toFIGS. 4-7 .

It should be understood that although FIG. 1 illustrates the variousmodules, such as the safety module 120, the position predictor 118, thetool controller 116 and the datastore 124, are included in the roboticsurgical device 114, one or more of these modules can be implemented indifferent ways within a robotic surgical system. For example, thefunctionality described above need not be separated into discretemodules, or some or all of such functionality may be located on acomputing device separate from the robotic surgical device 114, such asa central controlling device connected to the robotic surgical device114 directly or through a network and configured to control theoperations of the robotic surgical system 100.

In addition, upon detecting a safety issue, such as an out-of-viewsurgical being predicted to be in view, or a tool with a wrong type isloaded, a perceivable indication can be provided so that the personnelin the operating room, including the surgeon 102, can be notified of theissue. For example, a vibration can be generated, a special sound can beplayed, or an LED light on the robotic surgical device 114 or otherdevices in the robotic surgical system 100 can be turned on to indicatethe safety issue. A warning message or a highlight can be generated andpresented on the video monitor 108 of the surgeon console 104 to drawthe attention of the surgeon 102.

Furthermore, mechanisms can be implemented to enable the surgeon 102 orother authorized individuals to provide input to the safety checkperformed by the safety module 120. For example, a user interface can bedesigned and presented on the video monitor 108 along with the surgicalimages 132 to allow the surgeon 102 to flag certain surgical tools thatwill not be used by the surgeon 102 so that they can be ignored by thesafety module 120. Other mechanisms can be implemented to receive inputsfrom the surgeon 102 or authorized individuals.

Referring now to FIG. 4 where a block diagram illustrating aspects ofthe safety module 120 of the robotic surgical system 100 is shown. FIG.4 will be presented in conjunction with FIGS. 5A-5D where examplesimulated images of loaded surgical tools and identified surgical toolsare shown. As illustrated in FIG. 4 , the safety module 120 can includea tool detection module 402 configured to detect the surgical toolscaptured in the surgical images 132. The detection of the tools can beperformed by using object recognition or other image/video processingtechniques.

For example, a trained machine-learning (“ML”) technique, such as aconvolutional neural network, can be trained and utilized to recognizesurgical tools from images or videos. Although convolutional neuralnetworks are described herein, any suitable ML technique may be trainedto recognize different surgical tools from images or videos, such as along short-term memory (“LSTM”) technique, a dynamic time warping(“DTW”) technique, a hidden Markov model (“HMM”), etc., or combinationsof one or more of such techniques e.g., CNN-LSTM, CNN-HMM or MCNN(Multi-Scale Convolutional Neural Network).

The convolutional neural network can be trained, or fine-tuned if theneural network is pre-trained, by using a set of training samplesincluding surgical procedure images and corresponding labelingindicating whether a surgical tool exists in an image or not. After thetraining, the convolutional neural network can estimate whether a givenimage or a set of images contain surgical tools or not and, if so, wherethe surgical tools are positioned in the image.

The detected surgical tools can then be sent to a tool labeling module304, where the types of the identified surgical tools can be recognizedbased on the surgical images 132 containing the detected surgical tools.The tool types can include, but are not limited to, tools for graspingfor holding or retracting objects, such as forceps, graspers andretractors, tools for suturing and cutting, such as needle drivers,scalpels and scissors, and other tools that can be used during asurgery. Depending on the surgery being performed or to be performed,sub-type of the surgical tools can also be identified. For example, thetool labeling module 304 can identify a grasper tool as a fenestratedgrasper for grasping delicate tissue like peritoneum and bowel, or atraumatic grasper for securing thicker tissue or organs.

Similar to the tool detection module 302, the tool labeling module 304can employ machine learning or other object recognition techniques toestimate the tool type of the identified surgical tools. For example, aconvolutional neural network can be used to recognize the tool type foreach of the identified surgical tools. The convolutional neural networkcan be trained using training samples including images of surgicaltools, computer-generated images based on CAD files for the tools andtheir respective tool types. The training convolutional network can beapplied to the portion of the surgical images 132 that contain anidentified surgical tool to predict the tool type for the identifiedsurgical tool.

Alternatively, or additionally, identifying the tool type can beachieved through labeling the tool body. For example, a label or othervisible identifier carrying a serial number or anything representing theidentity of a surgical tool, such as a bar code or a quick response(“QR”) code, can be affixed to the body of the surgical tool, such asthe shaft of the surgical tool. The label can be identified from thesurgical images 132 and the serial number can be recognized orinterpreted from the identified label. The recognized serial number canthen be utilized to, for example, query a database of surgical tools, toidentify the corresponding tool type. Other types of identifiers may beemployed as well, such as an RFID tag affixed to the surgical tool andpositioned to be readable by the robotic surgical system, e.g., byaffixing the RFID tag at a location near the base of the surgical tool.

It should be appreciated that the methods presented above for detectingthe surgical tools and identifying the tool type from the surgicalimages 132 are for illustration only and should not be construed aslimiting. Other methods can also be employed to facilitate the detectionof the surgical too and the tool type. For instance, informationobtained through imaging mechanisms other than the camera 128 can alsobe utilized to facilitate the identification of the surgical tools.These imaging mechanisms can include, but are not limited to, theultrasound imaging, optical contrast imaging, fluorescent imaging and soon.

In one example, the camera 128 or the surgical tools 126 can be equippedwith an ultrasound probe for detecting objects such as lung tumor incertain type of surgeries. The tool detection module 402 and/or the toollabeling module 404 can access the ultrasound images taken by theultrasound probe to facilitate the detection of the surgical tool andthe tool type. These additional imaging mechanisms can be especiallyhelpful when the surgical tools 126 are hidden in the surgical images132, for example, by anatomy tissue. The detected tools can be utilizedto visualize the hidden tools, such as by outlining the specific toolsin the surgical images 132, which can be beneficial for tasks such asfind the one-to-one correspondence between the loaded surgical tools andthe identified surgical tools as will be discussed below.

The position and type information of the identified surgical tools canthen be sent to a tool matching module 406 to determine whether theinformation of the identified surgical tools match that of the loadedsurgical tools determined by the position predictor 118. To facilitatethe matching, a simulated camera image can be generated based on thepositions of the identified surgical tools and the loaded surgicaltools, the camera field of view and distortions. Example simulatedcamera images are shown in FIGS. 5A-5D, where both the identifiedsurgical tools 504 estimated from the surgical images 132 and the loadedsurgical tools 502 predicted by the position predictor 118 are shown. Inthese figures, the identified surgical tools 504 are illustrated insolid lines, and the loaded surgical tools 502 are illustrated in dottedlines.

In FIG. 5A, there are two identified surgical tools: a grasper 504A anda scalpel 504B, and two loaded surgical tools: a grasper 502A and ascalpel 502B. The tool matching module 406 can determine a one-to-onecorrespondence between the two sets of surgical tools. In someimplementations, the correspondence can be determined by choosing toolswith nearly matching positions in the simulated images and matching tooltypes. For example, the tool matching module 406 can determine that theposition of the loaded surgical tool 502A is closer to the position ofthe identified surgical tool 504A than the identified surgical tool504B, and that the type of the loaded surgical tool 502A matches that ofthe identified surgical tool 504A. The tool matching module 406 can thusdetermine that the identified surgical tool 504A and the loaded surgicaltool 502A correspond to each other. Similarly, the tool matching module406 can determine that the identified surgical tool 504B and the loadedsurgical tool 502B correspond to each other because the position of theloaded surgical tool 502B is closer to the position of the identifiedsurgical tool 504B than the identified surgical tool 504A and the typeof the loaded surgical tool 502B matches that of the identified surgicaltool 504B.

The difference between the positions of two surgical tools can becalculated, for example, as the Euclidean distance between center pointsof the two surgical tools. The center point of a surgical tool's endeffector can be selected as the center of gravity of the surgical toolor any other points that can be used to represent a surgical toolconsistently over the two sets of surgical tools. The difference betweenthe types of two surgical tools can be converted to a quantity with ascale that is comparable to the distance between the positions of thetwo surgical tools. As a result, the problem of finding the one-to-onecorrespondence between the loaded surgical tools 502 and the identifiedsurgical tools 504 can be formulated as an optimization problem such asfollows:

$\begin{matrix}{{\min\limits_{Q}{\sum\limits_{{({i,j})} \in Q}{\alpha_{1}{D\left( {P_{i},P_{j}} \right)}}}} + {\alpha_{2}{D\left( {T_{i},T_{j}} \right)}}} & (1)\end{matrix}$

where D(P_(i), P_(j)) is the difference between the position P_(i) ofthe identified surgical tool i and the position P_(j) of the loadedsurgical tool j projected onto the imagining plane of the camera 128;D(T_(i), T_(j)) is the difference between the type T_(i) of identifiedsurgical tool i and the type T_(j) of loaded surgical tool j. α₁ and α₂are two parameters that can be utilized to adjust the relativeimportance between the positional difference and the tool typedifference in the minimization problem. Q is a set containing a possiblecorrespondence between the identified surgical tools 504 and the loadedsurgical tools 502.

In the example shown in FIG. 5A, Q can be {(grasper 504A, grasper 502A),(scalpel 504B, scalpel 502B)}. That is, the identified grasper 504Acorresponds to the loaded grasper 502A and the identified scalpel 504Bcorresponds to the loaded scalpel 502B. In this example, (i,j) inEquation (1) can take the value of (grasper 504A, grasper 502A) and(scalpel 504B, scalpel 502B) when calculating the difference D(P_(i),P_(j)) and D (T_(i), T_(j)). An alternative value for Q in the exampleshown in FIG. 5A can be {(scalpel 504B, grasper 502A), (grasper 504A,scalpel 502B)}, i.e. the identified scalpel 504B corresponds to theloaded grasper 502A and the identified grasper 504A corresponds to theloaded scalpel 502B. The optimization problem shown in Equation (1) canfind the proper Q value, i.e. the proper correspondence between theidentified surgical tools 504 and the loaded surgical tools 502, byminimizing the difference between the two sets of surgical tools.

Additional requirements can be added to help identify the correspondencebetween the loaded surgical tools 502 and the identified surgical tools504. For example, relative position information, such as the grasper504A (or 502A) should be on the left side of the scalpel 504B (or 502B),can be used to limit the possible correspondences between the two setsof surgical tools. These additional information can reduce the searchspace of Q and can be formulated as one or more constraints of theoptimization problem in Equation (1). It should be understood that theoptimization problem described above is merely for illustrationpurposes, and should not be construed as limiting. Various methods offinding the correspondence between the loaded surgical tools and theidentified surgical tools can be utilized.

Alternatively, or additionally, a label or other visible identifier,such as a bar code or a QR code, representing an identification of asurgical tool can be affixed on the tool body. The identification of thesurgical tools can be read on the identified surgical tools 504 from thesurgical images 132 and interpreted to find the corresponding loadedsurgical tool 502. If visible identifiers are available on tool body,the optimization process described above can be utilized to verify thecorrespondence determined through the identifier or be used when thetool identifier cannot be read reliably from the surgical images 132.

The example shown in FIG. 5A has the same number of identified surgicaltools 504 and loaded surgical tools 502. It is, however, possible thatthe number of identified surgical tools 504 using the surgical images132 is different from the number of loaded surgical tools 502 determinedby the position predictor 118. In the example shown in FIG. 5B, thereare two loaded surgical tools: a grasper 502C and a scissor 502D, butonly one identified surgical tool: a grasper 504C. This can occur whenthe position predictor 118 mistakenly determine that the scissor tool502D is in the field of view of the camera 128 when it is not. In thiscase, the tool matching module 406 can determine that the number ofloaded surgical tools 502 and the identified surgical tools 504 do notmatch, and can perform further analysis to determine the unmatchedsurgical tool.

For example, the tool matching module 406 can first find out which ofthe loaded surgical tools 502C and 502D corresponds to the identifiedsurgical tool 504C, for example, by formulating and solving theoptimization problem of Equation (1) as discussed above. Afteridentifying that the loaded surgical tool 502C corresponds to theidentified surgical tool 504C, the tool matching module 406 can furtherretrieve information about the loaded surgical tool 502D that is missingfrom the surgical images 132. For instance, the tool matching module 406can determine that the loaded surgical tool 502D is a scissor tool andis loaded on the second robotic arm of the robotic surgical device 114.Based on this information, the tool matching module 406 can demand thatthe surgical tool carried on the second robotic arm to be disabled, forexample, by generating and sending safety instructions 136 to the toolcontroller 116. The tool controller 116 can disable the loaded surgicaltool 502D by, for example, preventing user input from being sent to thesurgical tool 502D; deactivating energy applied by the surgical tool502D, or locking the drive motors or joints of the surgical tool 502D.In this way, the out-of-view scissor tool 502D cannot be moved andoperated by the surgeon 102 until it is brought in the field of view ofthe camera 128, thereby preventing accidental injury to patient 130 bythe scissor tool 502D. To move a tool, such as the out-of-view scissortool 502D, into the field of view of the camera, the surgeon 102 canmove the camera to the location of the tool and then move the tool alongwith the camera back to the operating site.

FIG. 5C also shows an example where the number of the loaded surgicaltools 502 does not match the number of identified surgical tools 504. Inthis example, there are two identified surgical tools: a grasper 504Eand a scissor 504F, and one loaded surgical tool: a grasper 502E. Thiscan occur when the position predictor 118 mistakenly determined that thescissor tool is outside the field of view of the camera 128 and therebydisabled it preventing the surgeon from using it. Similar to the exampleshown in FIG. 5B, the tool matching module 406 can determine that thenumber of tools in the two sets of surgical tools does not match andfurther determine that the identified surgical tool 504E corresponds tothe loaded surgical tool 502E. Regarding the identified surgical tool504F, the tool matching module 406 can determine that the scissor toolcorresponding to the identified surgical tool 504F is in the field ofview of the camera 128 and is safe to use even though the positionpredictor 118 determines otherwise. As a result, the tool matchingmodule 406 can instruct, via safety instruction 136, the tool controller116 to enable the scissor tool. This can help to avoid the frustrationand confusion of the surgeon 102 caused by the prediction error of theposition predictor 118.

Sometimes, even if the number of loaded surgical tools 502 matches thenumber of the identified surgical tools 504, there can still be problemswith the surgical tools loaded by the robotic surgical device 114. FIG.5D illustrates such an example. In FIG. 5D, both the loaded surgicaltools 502 and the identified surgical tools 504 have two tools. However,the two tools that are close in position have different tool types, i.e.the loaded surgical tool 502G is a grasper tool and the identifiedsurgical tool 504G is a scissor tool. One possible reason for this isthat the tool controller 116 has loaded a wrong tool on the robotic arm.This can cause a serious problem in the surgery procedure, especiallywhen the shapes of the wrong tool and the correct tool are similar andthe surgeon cannot immediately recognize the difference in the surgicalimages 132 shown on the video monitor 108. Once the tool type mismatchis detected, the tool matching module 406 can generate a safetyinstruction 136 to disable the wrongly loaded surgical tool and providea warning to the surgeon 102.

As shown in FIG. 5A, even if the number of tools and the tool typesbetween the loaded surgical tools and the identified surgical toolsmatch, there can be discrepancy between the pose of an identifiedsurgical tool 504A and the pose of the corresponding loaded surgicaltool 502A. As discussed above with respect to FIG. 3B, such adiscrepancy can be the result of deflection on the surgical tool causedby forces unknown to the position predictor 118. If the discrepancy islarge, it can mean that an excessive force has been applied by thesurgical tool on the patient body without the surgeon realizing it. Toaddress this problem, the safety module 120 can include a tool poseestimation module 408 to estimate the tool pose of the identifiedsurgical tools 504 relative to the camera 128. Such estimated tool poseof a surgical tool can be compared with a predicted tool pose of thesurgical tool. The difference between the estimated and predicted toolpose can be utilized to determine the amount of forces applied by thesurgical tool causing the deflection of the tool.

The tool pose estimation can be performed based on the surgical images132. For example, object recognition techniques can be utilized toidentify the pose of a surgical tool in the image, including theposition of the surgical tool and the orientation of the surgical toolrelative to the camera in a three-dimensional space. For example, aneural network trained to recognize the surgical tool and the pose ofthe surgical tool can be utilized to identify the tool pose.Alternatively, or additionally, the tool pose estimation module 408 candetermine the pose of an identified surgical tool 504 by comparing theidentified surgical tool 504 to a datastore 410 containing images ofsurgical tools in various poses. The tool pose can be determined ifthere is a match between the image of the identified surgical tool 504and an image in the datastore 410.

In addition to estimate the tool pose using the two-dimensional images,the knowledge of the tool type and the tool geometry, such as the shaftwidth or head length, can be utilized to estimate the depth of thesurgical tool relative to the camera 128 so that the tool pose in thethree-dimensional space can be determined. Furthermore, in scenarioswhere the camera 128 is a stereoscopic camera or the robotic surgicaldevice 114 is equipped with a stereoscopic camera, the depth informationof the surgical tool can be estimated via triangulation of correspondingpixels on the stereo images. Additional fiducials could also be added tothe tool shaft to give visual reference points for measurement. Forexample, a mark or other type of labels, can be added on the tool shaftfor every centimeter to provide a visual reference points for theestimation of the tool pose. Various other methods can be utilized toestimate the pose of the identified surgical tools 134 in the surgicalimages 132.

The estimated pose 414 of the identified surgical tools 504 can then tobe sent to the tool matching module 406 to estimate the force exerted bythe surgical tool. The tool matching module 406 can compare theestimated pose 414 of a identified surgical tool 504 with a predictedpose of a corresponding loaded surgical tool 502. As discussed above, akinematic chain model of the robotic arm holding the loaded surgicaltool 502 and a kinematic chain model of the robotic arm holding thecamera 128 can be utilized to predicted the pose of the loaded surgicaltool 502. Through the comparison, the tool matching module 406 cancalculate the difference between the estimated tool pose 414 and thepredicted tool pose.

The tool matching module 406 can then estimate the force of the surgicaltool based on the difference between the estimated and actual tool posesas well as other information, such as the stiffness of the surgical tooland the associated robotic arm, the location of the applied force alongthe kinematic chain. The stiffness of the surgical tool and the roboticarm can be determined by the geometry and materials of the surgical tooland the robotic arm holding the surgical tool. The analysis can beperformed for each of the identified surgical tools in the field of viewof the camera that have a corresponding loaded surgical tool 502. Itshould be noted that this estimation is based on an assumption that thedifference in the tool poses of an identified surgical tool 504 and itscorresponding loaded surgical tool 502 is the result of tool forcesdeflecting the surgical tool. If the deflection, thus the tool posedifference, is observed for all the surgical tools under analysis, it islikely that the camera 128 has been deflected causing the global effectof common deflection on all surgical tools. In that case, the effect ofthe camera deflection can be subtracted from the tool pose estimation ofthe identified surgical tool 504 when estimating the tool force.

The estimated force of the surgical tool can then be compared with aforce threshold. If the estimated force is larger than a threshold, thetool matching module 406 can generate a warning message to notify thesurgeon 102 about the excessive tool force. For example, the surgicaltool that is exerting excessive force can be flagged or highlighted inthe surgical images 132 shown on the video monitor 108. If the estimatedforce is too excessive, such as higher than an even higher forcethreshold, the tool matching module 406 can cause the robotic surgicaldevice 114 to be paused, for example, by sending the safety instructions136 to the tool controller 116 to request pausing the robotic surgicaldevice 114 to temporarily preventing the surgical procedure to continue.The robotic surgical device 114 can resume its normal operations afterthe surgical tool exerting highly excessive force is withdrawn, moved toanother position, or changed in other ways to reduce its force, such asvia a reduction in proximal motor torques.

It should be understood that while the above process describes simulatedcamera images, such as those shown in FIGS. 5A-5D, generating thesesimulated camera images are optional and the technology described hereindoes not require those simulated images.

Referring now to FIG. 6 , where an example method 600 improving roboticsurgical safety via video processing is presented. The example method600 will be discussed with respect to the example system 100 shown inFIG. 1 , but may be employed according to any suitable system accordingto this disclosure.

At block 602, the safety module 120 obtains or accesses surgical images132 of an ongoing surgical procedure from, for example, from a real-timevideo feed from an endoscopic camera 128.

At block 604, the safety module 120 identifies surgical tools used inthe surgical procedure using the surgical images 132. As discussedabove, the identification can be performed using trained ML techniquesor other object recognition or image/video processing techniques. In oneexample, the identification of the surgical tool can be performed byapplying a convolutional neural network trained for recognizing surgicaltools from images or videos onto the surgical images 132. The output ofthe neural network can indicate whether a surgical image 132 or a groupof surgical images 132 contain surgical tools and if so, the positionsof the surgical tools in the two-dimension surgical images 132.

In addition to the surgical images 132, other imaging mechanisms can beutilized to facilitate the identification of surgical tools in thesurgical images 132, such as ultrasound imaging, optical contrastimaging, fluorescent imaging and so on. These additional imagingmechanism can be useful to identify surgical tools when they are noteasily recognized from the 132, such as being blurred or be hidden bythe patient tissue.

The identified surgical tools can then be analyzed to identify the typesof the respective identified surgical tools at block 606. The tool typecan be determined by employing a neural network training to recognizevarious types of surgical tools. Images or image patches containingsurgical tools can be feed into the neural network and the output of theneural network can indicate the type of the surgical tool contained inthe input image. For example, the neural network can be trained andutilized to determine the tool types including, but not limited to,tools for grasping for holding or retracting objects, such as forceps,graspers and retractors, tools for suturing and cutting, such as needledrivers, scalpels and scissors, and other tools that can be used duringa surgery. Depending on how the neural network is trained, it can alsobe utilized to identify sub-types of the surgical tools. For example,the neural network can be utilized to identify a grasper tool as afenestrated grasper for grasping delicate tissue like peritoneum andbowel, or a traumatic grasper for securing thicker tissue or organs.

In some scenarios, some of the surgical tools can have a label or othervisible identifier affixed to the tool body, such as the shaft of thesurgical tool. If the label can be recognized from the surgical images132, then the surgical tools and their respective tool types can beidentified by interpreting the label and referencing to a surgical tooldatabase.

At block 608, the position predictor 118 can generate the loaded toolinformation 122 including the types, positions/poses and otherinformation of the loaded surgical tools 502 at the time when thesurgical images 132 are captured. The position predictor 118 can obtaindata about the loaded surgical tools 502, such as the tool types and therobotic arms for the respective loaded surgical tool 502, from the toolcontroller 116. Such information may be provided when each of thesurgical tools is loaded in the robotic surgical system 100, or may beobtained based on one or more identifiers provided on the surgicaltool(s). Based on the information, the position predictor 118 canpredict the position and pose of each loaded surgical tool 502 withrespect to the camera 128 based on a kinematic chain model of therobotic arm holding the corresponding surgical tool and a kinematicchain model of the robotic arm holding the camera 128. The predictedthree-dimension pose information of a loaded surgical tool 502 can betransformed into a two-dimensional position information of the loadedsurgical tool 502 by projecting the position and pose information ontothe imaging plane of the camera 128 so that the position of the loadedsurgical tool 502 can be compared with the position of the identifiedsurgical tool 504 in the surgical images 132. The predicted pose and/orposition of the loaded surgical tools 502 can be included in the loadedtool information 122 along with the tool types and be sent to the safetymodule 120 for comparison.

At block 610, the safety module 120 tries to determine a one-to-onecorrespondence between the loaded surgical tools 502 and the identifiedsurgical tools 504. As discussed above in detail, the correspondence canbe determined by solving an optimization problem, such as thatformulated in Equation (1), through iterative operations to find a setof correspondence between the two sets of surgical tools that minimizethe overall distance between the loaded surgical tools 502 and theidentified surgical tools 504. Various other methods for find thecorrespondence can also be utilized.

After finding the correspondence, the safety module 120 can determine,at block 612, whether the number of loaded surgical tools 502 matchesthe number of the identified surgical tool 504. If the two numbers donot match, the safety module 120 can identify those loaded surgicaltools 502 that are not in the identified surgical tools 504. These toolscan potentially cause injuries to patient because they can be operatedby the surgeon 102 but are not in the field of view of the camera 128.The safety module 120 can thus instruct the tool controller 116 todisable these tools.

At block 616, the safety module 120 also identifies those tools that arein identified surgical tool 504 but are not in the loaded surgical tool502. These tools can cause confusion and frustration of the surgeon 102because they are in the field of view of the camera 128 but are disabledbecause they are predicted to be outside the camera view by the positionpredictor 118. The safety module 120 can instruct the tool controller116 to enable these tools so that surgeon 102 can use them when needed.

For those loaded surgical tool 502 that have a corresponding identifiedsurgical tool 504, the safety module 120 further determines, at block618, if the type of loaded surgical tool 502 matches the type of itscorresponding identified surgical tool 504. If the tool type does notmatch, the safety module 120 can instruct the tool controller 116 todisable the loaded surgical tool 502 because it is likely that a wrongsurgical tool has been loaded. For the rest of the loaded surgical tools502, i.e. loaded surgical tools 502 that have a corresponding identifiedsurgical tool 504 with a matching tool type, the safety module 120 canperform a safety check on the tool force at block 622 to determine ifany excessive force has been applied by the loaded surgical tool 502.Details regarding the operations at block 622 will be presented belowwith respect to FIG. 7 .

At block 624, the safety module 120 determines if it will continue tomonitor the safety of the robotic surgical device 114. If so, the safetymodule 120 repeats the method as described above for a next period oftime of the surgical procedure. If not, the method ends at block 626.

It should be appreciated that the description of the method 600 of FIG.6 is merely one example sequence, and that one or more blocks of themethod 600 may be performed in different orders or may be omittedentirely. For example, generating the loaded tool information 122 atblock 608 can be performed in parallel with the identification of theidentified surgical tool 504 at blocks 602-606. In another example, thesafety module 120 may omit the safety check on tool force in block 622,for example, in order to speed up the process so that a real-time safetyfeedback can be provided.

Referring now to FIG. 7 , where an example method 700 for estimating theforce exerted by a surgical tool is illustrated. The example method 700will be discussed with respect to the example system 100 shown in FIG. 1, but may be employed according to any suitable system according to thisdisclosure. At block 702, the safety module 120 estimates the pose of anidentified surgical tool 504 in the surgical images 132. As discussedabove, a pose of a surgical tool includes the position of the surgicaltool and the orientation of the surgical tool relative to the camera ina three-dimensional space. The estimation of the pose can be performedby applying one or more trained ML techniques or other objectrecognition techniques on the surgical images 132, such as by using aneural network model trained to recognize surgical tool poses.Alternatively, or additionally, the pose of an identified surgical tool504 can be determined by comparing the identified surgical tool 504 to adatastore 410 containing images of surgical tools in various poses.Other methods and information can also be utilized to determine the poseof the identified surgical tool 504 in the three dimensional space, suchas the tool type and the tool geometry, stereo images of the tool, andso on.

At block 704, the safety module 120 can access the pose of the loadedsurgical tool 502 that is predicted by the position predictor 118 asdescribed in block 608 of the method 600. Then at block 706, the safetymodule 120 calculates the difference between the estimated pose of theidentified surgical tool 504 and the predicted pose of the loadedsurgical tool 502. Such a difference is then utilized to determine thetool force excreted by the loaded surgical tool 502 at block 708. Asdiscussed above, the difference in the tool poses of the loaded surgicaltool 502 and identified surgical tool 504 can be the result of the forcedeflecting the loaded surgical tool 502. As such, the difference in toolposes, along with other information such as the stiffness of the tool,can be utilized to determine that force that caused the tool deflection.

At block 710, the estimated tool force is compared with a forcethreshold. If the estimated tool force is higher than the forcethreshold, a warning message can be generated and sent to the surgeon102 at block 712 to notify him to take proper actions to reduce the toolforce. At block 714, the safety module 120 determines if there are moreloaded surgical tools 502 for which the tool force is to be estimated.If yes, the safety module 120 repeats the method 700 as described above.If no more loaded surgical tools 502 to be analyzed, method 700 ends atblock 716.

It should be appreciated that while the blocks of the method 1500 aredepicted and described in a particular order, no such ordering isrequired. For example, blocks 702 and 704 may be performed in any orderor substantially simultaneously. In addition, the force estimation fordifferent loaded surgical tools 502 can be determined substantiallysimultaneously.

Referring now to FIG. 8 , FIG. 8 shows an example computing device 800suitable for use in example systems or methods for improving roboticsurgical safety via video processing. The example computing device 800includes a processor 810 which is in communication with the memory 820and other components of the computing device 800 using one or morecommunications buses 802. The processor 810 is configured to executeprocessor-executable instructions stored in the memory 820 to performsecurity check of the robotic surgical device 114 according to differentexamples, such as part or all of the example methods 600, 700 describedabove with respect to FIGS. 6 and 7 . The computing device, in thisexample, also includes one or more user input devices 870, such as akeyboard, mouse, touchscreen, microphone, etc., to accept user input.The computing device 800 also includes a 860 display to provide visualoutput to a user.

The computing device 800 can include or be connected to one or morestorage devices 830 that provides non-volatile storage for the computingdevice 800. The storage devices 830 can store system or applicationprograms and data utilized by the computing device 800, such as modulesimplementing the functionalities provided by the safety module 120 andthe position predictor 118. The storage devices 730 might also storeother programs and data not specifically identified herein.

The computing device 800 also includes a communications interface 840.In some examples, the communications interface 840 may enablecommunications using one or more networks, including a local areanetwork (“LAN”); wide area network (“WAN”), such as the Internet;metropolitan area network (“MAN”); point-to-point or peer-to-peerconnection; etc. Communication with other devices may be accomplishedusing any suitable networking protocol. For example, one suitablenetworking protocol may include the Internet Protocol (“IP”),Transmission Control Protocol (“TCP”), User Datagram Protocol (“UDP”),or combinations thereof, such as TCP/IP or UDP/IP.

While some examples of methods and systems herein are described in termsof software executing on various machines, the methods and systems mayalso be implemented as specifically configured hardware, such asfield-programmable gate array (FPGA) specifically to execute the variousmethods. For example, examples can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or in acombination thereof. In one example, a device may include a processor orprocessors. The processor comprises a computer-readable medium, such asa random access memory (RAM) coupled to the processor. The processorexecutes computer-executable program instructions stored in memory, suchas executing one or more computer programs. Such processors may comprisea microprocessor, a digital signal processor (DSP), anapplication-specific integrated circuit (ASIC), field programmable gatearrays (FPGAs), and state machines. Such processors may further compriseprogrammable electronic devices such as PLCs, programmable interruptcontrollers (PICs), programmable logic devices (PLDs), programmableread-only memories (PROMs), electronically programmable read-onlymemories (EPROMs or EEPROMs), or other similar devices.

Such processors may comprise, or may be in communication with, media,for example computer-readable storage media, that may store instructionsthat, when executed by the processor, can cause the processor to performthe steps described herein as carried out, or assisted, by a processor.Examples of computer-readable media may include, but are not limited to,an electronic, optical, magnetic, or other storage device capable ofproviding a processor, such as the processor in a web server, withcomputer-readable instructions. Other examples of media comprise, butare not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip,ROM, RAM, ASIC, configured processor, all optical media, all magnetictape or other magnetic media, or any other medium from which a computerprocessor can read. The processor, and the processing, described may bein one or more structures, and may be dispersed through one or morestructures. The processor may comprise code for carrying out one or moreof the methods (or parts of methods) described herein.

The foregoing description of some examples has been presented only forthe purpose of illustration and description and is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Numerous modifications and adaptations thereof will be apparent to thoseskilled in the art without departing from the spirit and scope of thedisclosure.

Reference herein to an example or implementation means that a particularfeature, structure, operation, or other characteristic described inconnection with the example may be included in at least oneimplementation of the disclosure. The disclosure is not restricted tothe particular examples or implementations described as such. Theappearance of the phrases “in one example,” “in an example,” “in oneimplementation,” or “in an implementation,” or variations of the same invarious places in the specification does not necessarily refer to thesame example or implementation. Any particular feature, structure,operation, or other characteristic described in this specification inrelation to one example or implementation may be combined with otherfeatures, structures, operations, or other characteristics described inrespect of any other example or implementation.

Use herein of the word “or” is intended to cover inclusive and exclusiveOR conditions. In other words, A or B or C includes any or all of thefollowing alternative combinations as appropriate for a particularusage: A alone; B alone; C alone; A and B only; A and C only; B and Conly; and A and B and C.

That which is claimed is:
 1. A method that includes one or moreprocessing devices performing operations comprising: identifying asurgical tool used by a robotic surgical device in a surgical procedureusing one or more images of the surgical procedure; estimating a pose ofthe surgical tool based, at least in part, upon the one or more images;determining a difference between the estimated pose of the surgical tooland a predicted pose of the surgical tool predicted through a kinematicchain model of the robotic surgical device; calculating a force exertedby the surgical tool based, at least in part, upon the difference of theestimated pose and the predicted pose of the surgical tool; determiningthat the force is higher than a threshold; and causing the force of thesurgical tool to be reduced.
 2. The method of claim 1, whereinestimating the pose of the surgical tool based on the one or more imagescomprises at least one of matching an image of the surgical tool with adatabase of tool pose images, or predicting the pose of the surgicaltool by applying a neural network trained to identify tool poses on theone or more images.
 3. The method of claim 1, wherein estimating thepose of the surgical tool comprises estimating a depth of the surgicaltool relative to a camera capturing the one or more images.
 4. Themethod of claim 3, wherein the depth of the surgical tool is estimatedbased on one or more of a geometry of the surgical tool, one or morefiducials on the surgical tool, or one or more stereo images of thesurgical tool.
 5. The method of claim 1, wherein the force of thesurgical tool is calculated further based on one or more of a geometryof the surgical tool or a material of the surgical tool.
 6. The methodof claim 1, wherein identifying the surgical tool using the one or moreimages comprises: estimating a position of the surgical tool; andrecognizing a type of the surgical tool using the one or more images. 7.The method of claim 1, wherein identifying the surgical tool from theone or more images comprises: identifying an identifier visible from theone or more images; and interpreting the identifier to identify thesurgical tool.
 8. A non-transitory computer-readable medium comprisingprocessor-executable instructions to cause a processor to: identify asurgical tool used by a robotic surgical device in a surgical procedureusing one or more images of the surgical procedure; estimate a pose ofthe surgical tool based, at least in part, upon the one or more images;determine a difference between the estimated pose of the surgical tooland a predicted pose of the surgical tool predicted through a kinematicchain model of the robotic surgical device; calculate a force exerted bythe surgical tool based, at least in part, upon the difference of theestimated pose and the predicted pose of the surgical tool; determinethat the force is higher than a threshold; and cause the force of thesurgical tool to be reduced.
 9. The non-transitory computer-readablemedium of claim 8, wherein estimating the pose of the surgical toolbased on the one or more images comprises at least one of matching animage of the surgical tool with a database of tool pose images, orpredicting the pose of the surgical tool by applying a neural networktrained to identify tool poses on the one or more images.
 10. Thenon-transitory computer-readable medium of claim 8, wherein estimatingthe pose of the surgical tool comprises estimating a depth of thesurgical tool relative to a camera capturing the one or more images. 11.The non-transitory computer-readable medium of claim 10, wherein thedepth of the surgical tool is estimated based on one or more of ageometry of the surgical tool, one or more fiducials on the surgicaltool, or one or more stereo images of the surgical tool.
 12. Thenon-transitory computer-readable medium of claim 8, wherein the force ofthe surgical tool is calculated further based on one or more of ageometry of the surgical tool or a material of the surgical tool. 13.The non-transitory computer-readable medium of claim 8, whereinidentifying the surgical tool using the one or more images comprises:estimating a position of the surgical tool; and recognizing a type ofthe surgical tool using the one or more images.
 14. The non-transitorycomputer-readable medium of claim 8, wherein identifying the surgicaltool from the one or more images comprises: identifying an identifiervisible from the one or more images; and interpreting the identifier toidentify the surgical tool.
 15. A computing device comprising: aprocessor; and a non-transitory computer-readable medium havingprocessor-executable instructions stored thereupon, which, when executedby the processor, cause the processor to: identify a surgical tool usedby a robotic surgical device in a surgical procedure using one or moreimages of the surgical procedure; estimate a pose of the surgical toolbased, at least in part, upon the one or more images; determine adifference between the estimated pose of the surgical tool and apredicted pose of the surgical tool predicted through a kinematic chainmodel of the robotic surgical device; calculate a force exerted by thesurgical tool based, at least in part, upon the difference of theestimated pose and the predicted pose of the surgical tool; determinethat the force is higher than a threshold; and cause the force of thesurgical tool to be reduced.
 16. The non-transitory computer-readablemedium of claim 15, wherein estimating the pose of the surgical toolbased on the one or more images comprises at least one of matching animage of the surgical tool with a database of tool pose images, orpredicting the pose of the surgical tool by applying a neural networktrained to identify tool poses on the one or more images.
 17. Thenon-transitory computer-readable medium of claim 15, wherein estimatingthe pose of the surgical tool comprises estimating a depth of thesurgical tool relative to a camera capturing the one or more images. 18.The non-transitory computer-readable medium of claim 17, wherein thedepth of the surgical tool is estimated based on one or more of ageometry of the surgical tool, one or more fiducials on the surgicaltool, or one or more stereo images of the surgical tool.
 19. Thenon-transitory computer-readable medium of claim 15, wherein the forceof the surgical tool is calculated further based on one or more of ageometry of the surgical tool or a material of the surgical tool. 20.The non-transitory computer-readable medium of claim 15, whereinidentifying the surgical tool using the one or more images comprises:estimating a position of the surgical tool; and recognizing a type ofthe surgical tool using the one or more images.